Skip to content

feat(scope): read the targets of web and TLS tools (step 5, PR 4b) - #20

Merged
mkhellat merged 4 commits into
mainfrom
feat/scope-web
Nov 12, 2026
Merged

feat(scope): read the targets of web and TLS tools (step 5, PR 4b)#20
mkhellat merged 4 commits into
mainfrom
feat/scope-web

Conversation

@mkhellat

@mkhellat mkhellat commented Nov 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Owner

Phase 10, step 5 plan, PR 4b: the scope guard reads the targets of web and TLS tools. Until now a network step running any of them was refused as unparsed.

Tools: curl, wget, whatweb, nikto, gobuster (dir, vhost, dns), ffuf, feroxbuster, sslscan, testssl.sh.

What counts as a target

  • URL hosts. They're read by a strict grammar of codebud's own. A URL that two parsers could read differently is refused: a second @, a backslash, or a %-escape in the host.
  • Hosts a tool contacts on the way. Proxies, curl's --resolve addresses and --connect-to hosts, DNS servers, DoH servers, and testssl's --ip.
  • Proxies on this machine are not targets. An intercepting proxy like Burp passes each request on to the URL's own host, and that host is judged.

What is refused, and why

  • Following redirects: curl -L, -r in gobuster, ffuf and feroxbuster, nikto -followredirects. A redirect goes wherever the server says. wget and whatweb follow redirects by default, so they need --max-redirect=0 and --follow-redirect=never.
  • A fuzz keyword before the path: in the scheme, userinfo or host. This covers FUZZ and custom -w file:KW keywords. The wordlist is never read, so a word like evil.example/# would become the host. This departs from the plan, which would have allowed FUZZ.<scoped domain>: that is unsafe for the same reason.
  • Unsafe feroxbuster wordlist words. feroxbuster joins each word to the URL, so its wordlist is read first. Words that would leave the host (//x, \\x, https:x) refuse the step, and -w is required.
  • Inputs from elsewhere:
    • option and config files: curl -K, wget -e/--config (-e robots=off is allowed), nikto -config, ffuf -request/-config;
    • ffuf -input-cmd;
    • target files, stdin and mass testing (testssl --file, --mx);
    • a ferox-config.toml or nikto.conf in the project, because a step could have written it.
  • Things that can't be judged:
    • URL globbing without -g;
    • schemes other than http(s) and ftp(s);
    • gobuster's fuzz, s3, gcs and tftp modes;
    • local sockets;
    • unknown options.

gobuster dns only looks names up, so it is passive: out of scope it asks instead of refusing.

Verification

  • make check: 1632 passed, including 93 new tests in tests/test_scope_web.py.
  • make distcheck passes; make info and makeinfo HTML build without warnings; ruff check and format are clean.

@mkhellat
feat(scope): read the targets of web and TLS tools
94679c0 
Phase 10, step 5 plan, PR 4b.  The engagement scope guard now reads
the targets of curl, wget, whatweb, nikto, gobuster, ffuf, feroxbuster,
sslscan and testssl.sh, so these may run under an engagement instead
of being refused as unparsed.

A URL's host is read by a strict grammar of codebud's own; a URL two
parsers could read differently (a second @, a backslash, a %-escape in
the host) is refused.  Proxies, curl --resolve and --connect-to hosts,
DNS and DoH servers are targets; a proxy on this machine (Burp) is not,
since it passes requests on to the URL's own host.

Following redirects is refused (curl -L, -r of the fuzzers, nikto
-followredirects); wget and whatweb, which follow them by default,
need --max-redirect=0 and --follow-redirect=never.  A fuzz keyword
(FUZZ, or one named in -w file:KEYWORD) is allowed in the path, query
and headers, never before the path: the wordlist is not read, and a
word like evil.example/# would become the host.  feroxbuster joins
words to the URL, so its wordlist is read and a word that would leave
the host (//x, https:x) refuses it.  Config files a step could write in
the project (ferox-config.toml, nikto.conf), option files (curl -K,
wget -e/--config, nikto -config, ffuf -request), target files and
mass-testing modes are refused, as are gobuster's fuzz, s3, gcs and
tftp modes.  gobuster dns is passive: out of scope it asks.

* src/codebud_sec/scope.py (_host_port, _url, _proxy, _loopback)
(_no_local_config, _need_targets, _wordlist_problem): New helpers.
(_curl, _wget, _whatweb, _nikto, _gobuster, _ffuf, _feroxbuster)
(_sslscan, _testssl): New parsers.
(PARSERS): Add them.
* tests/test_scope_web.py: New file.
* tests/test_scope.py (test_an_unparsed_network_tool_is_refused):
Use socat; curl has a parser now.
* docs/chapters/architecture.texi, docs/chapters/development.texi:
Describe them.
@mkhellat

mkhellat commented Nov 12, 2026

Copy link
Copy Markdown
Owner Author

Review summary: changes requested

One full sweep over 94679c0: conditions → scenarios → each checked against the code → De Morgan.

Conditions. For a network step naming a parsed tool, the step is refused ⇔ any of these:

  • the targets can't be read (unknown option, ambiguous URL, a file or command input, redirects followed, a fuzz keyword before the path);
  • an active tool has a target out of scope;
  • the window is closed.

A passive tool with a target out of scope (gobuster dns) asks. The invariant behind it: the hosts the tool will contact ⊆ the hosts read from its words.

Correct, verified (93 tests; each refusal checked to fire for its stated reason, not by accident):

  • URL hosts are read by codebud's own grammar; a@b@host, a backslash, or % in the host is refused.
  • Proxies, --resolve, --connect-to, DNS and DoH servers, and testssl --ip are all judged.
  • Following redirects is refused everywhere, and wget and whatweb must turn it off.
  • A fuzz keyword in the scheme, userinfo or host is refused, including custom -w file:KW keywords.
  • Config, option, target and request files, and stdin are refused; so are gobuster's fuzz, s3, gcs and tftp modes, and URL globbing without -g.
  • A loopback proxy is exempt; the URL host is still judged.

Defects (each reproduced as ALLOWED on 94679c0) — four ways the invariant breaks, plus one false refusal:

  1. Environment assignments before the tool. The proxy or config they set is invisible to the parser:

    • env HTTPS_PROXY=http://evil.example.org:8080 curl … (Go tools honour it too);
    • env WGETRC=./x wget … and env HOME=. curl …, which make the tool read a config a step can write.

    Fix: refuse an assignment before a parsed tool, except a few harmless names (LANG, LC_*, TZ, TERM, NO_COLOR, COLUMNS).

  2. sslscan takes values only as --opt=value. With --certs c.pem host, sslscan does not read c.pem as the certificate; the parser does. --xmpp-server is a flag, but the parser takes the next word as its value. Either way, the parser can judge a different host from the one sslscan scans. Fix: value options only in = form, --xmpp-server as a flag, and exactly one host operand unless --targets= is given.

  3. testssl.sh --phone-out contacts third-party CRL and OCSP servers, and it was parsed as taking a value. Fix: refuse it.

  4. gobuster dir -u https://app.example.com with no path: the URL and the word are concatenated. Unless a path keeps them apart, a word like @evil.example.org falls into the authority. Fix: dir mode needs a URL with a path (…/).

  5. The feroxbuster wordlist check doesn't follow the URL standard. It strips leading C0 controls and removes tabs and newlines anywhere, so \x01//evil and /\t/evil become //evil and pass. It is also too broad: javascript:void(0) is refused, though only special schemes reach a host. Fix: normalise as WHATWG does, then refuse //, a backslash pair, and http(s):, ftp: or ws(s):.

Verdict: changes requested. The fixes come as new commits, then a re-review.

@mkhellat
fix(scope): refuse environment assignments before a network tool
a510d52 
Review of #20.  env HTTPS_PROXY=... curl, env WGETRC=./x wget or
env HOME=. curl give the tool a proxy, or a config file a step can
write, that its words do not show, so the guard judged the wrong hosts.
An assignment before a parsed tool is now refused, except for locale,
time zone and terminal names.

* src/codebud_sec/scope.py (_ASSIGNMENT, _HARMLESS_ENV): New.
(targets_of): Refuse other assignments before a parsed tool.
* tests/test_scope_web.py
(test_environment_assignments_before_a_tool_are_refused)
(test_harmless_environment_assignments_are_fine): New tests.
@mkhellat
fix(scope): read sslscan as it reads itself; refuse testssl --phone-out
e277f24 
Review of #20.  sslscan takes values only as --opt=value, and
--xmpp-server is a flag, so taking the next word as a value judged a
host it does not scan (sslscan --certs evil.example.org app...).
Values are now accepted only after =, --xmpp-server is a flag, and one
host is taken unless --targets=file lists them.  testssl.sh
--phone-out contacts the CRL and OCSP servers of third parties; it was
read as taking a value, and is now refused.

* src/codebud_sec/scope.py (_list_file): Take a spec function.
(_SSLSCAN_VALUE): Remove.
(_SSLSCAN_PATTERNS): Values only after =; --xmpp-server a flag.
(_sslscan): One host, or --targets=file.
(_TESTSSL_VALUE, _TESTSSL_FLAGS): --phone-out is a flag.
(_testssl): Refuse it.
* tests/test_scope_web.py (test_sslscan_is_read_as_sslscan_reads_it)
(test_sslscan_values_after_equals_are_fine)
(test_testssl_phone_out_is_refused): New tests.
@mkhellat
fix(scope): keep appended words out of the host
bfb06cb 
Review of #20.  gobuster dir appends each word to its URL; without a
path, a word like @evil.example.org lands in the authority, so dir mode
now needs a URL with a path.  feroxbuster joins words to its URL, and
the wordlist check missed words the URL standard cleans up into //host
(leading control characters, tabs and newlines anywhere, as in
/<TAB>/evil); it now cleans up as WHATWG URL does first.  It refused
words with any scheme, though only special schemes reach a host, so
javascript: and mailto: lines no longer refuse a wordlist.

* src/codebud_sec/scope.py (_url): New argument need_path.
(_gobuster): Use it in dir mode.
(_JOIN_ESCAPE): Two slashes or backslashes, or a special scheme.
(_C0_AND_SPACE): New.
(_wordlist_problem): Clean up each word as the URL standard does.
* tests/test_scope_web.py
(test_gobuster_dir_needs_a_path_to_keep_words_out_of_the_host)
(test_feroxbuster_wordlist_is_judged_after_the_url_cleanup)
(test_feroxbuster_wordlist_words_that_reach_no_host_are_fine): New.
@mkhellat

mkhellat commented Nov 12, 2026

Copy link
Copy Markdown
Owner Author

Re-review summary: fixed, ready to merge once CI passes

Second sweep, over bfb06cb.

Finding Fix Verified
1. Environment assignments before the tool (env HTTPS_PROXY=…, WGETRC=, HOME=) a510d52: an assignment before a parsed tool is refused, except LANG, LC_*, TZ, TERM, NO_COLOR, COLUMNS 6 cases allowed before, refused now; harmless names still run
2. sslscan read differently from sslscan's own parsing e277f24: values only as --opt=value, --xmpp-server a flag, one host unless --targets=file 4 cases refused; the = forms are read
3. testssl.sh --phone-out contacts third parties e277f24: refused (it is a flag, now parsed as one) was allowed, now refused
4. gobuster dir -u https://host without a path bfb06cb: dir mode needs a URL with a path was allowed, now refused; …/ still runs
5. feroxbuster wordlist check against the URL standard bfb06cb: words are cleaned up as WHATWG does (C0 controls and spaces stripped, tabs and newlines removed), then //, backslash forms and special schemes are refused \x01//…, /\t/…, \/…, HTTPS:x refused; javascript: and mailto: lines are fine now

Erring strict, documented in the code:

  • Long-option abbreviations (wget, nikto, whatweb) are refused as unknown options.
  • Multi-value clap options (feroxbuster -x php txt) need commas.
  • Attached short values (wget -qO-) need separate words.

1651 tests pass, ruff is clean, and make distcheck passes. Verdict: correct. I'll merge with a merge commit once CI passes on all jobs, macOS included.

@mkhellat

mkhellat commented Nov 12, 2026

Copy link
Copy Markdown
Owner Author

Merging. All five review findings are fixed and re-verified: environment assignments, sslscan parsing, testssl --phone-out, the gobuster dir path, and the feroxbuster wordlist clean-up. CI is green on all five jobs at bfb06cb, macOS included. main has not moved since the branch was cut, so no combined test was needed. A merge commit keeps the four commits as written.

@mkhellat
mkhellat merged commit 190b59f into main Nov 12, 2026
5 checks passed
@mkhellat
mkhellat deleted the feat/scope-web branch November 12, 2026 11:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mkhellat